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The VSA College Portrait demonstrated that public universities 
are committed to greater transparency of information and a 
willingness to respond to calls for accountability through assessing 
and reporting learning outcomes. 


This project would not have been possible without the cooperation of focus group 
participants, survey respondents, and the assistance and feedback provided by Christine 
Keller and Wendell Hall of the VSA. 


Transparency & Accountability 


2 


Executive Summary 


a 
Transparency & Accountability: An Evaluation of the VSA College Portrait 


The Voluntary System of Accountability (VSA) is a vehicle for public four-year universities to report 
comparable information about the undergraduate student experience via the College Portrait, a 
common web reporting template. The National Institute for Learning Outcomes Assessment (NILOA) 
was asked by the VSA to evaluate the effectiveness of the student learning outcomes pilot project within 
the College Portrait. The evaluation was conducted over a five-month period, October 2011 through 
February 2012, drawing on a variety of data sources. These included focus groups, interviews with 
leaders from the policy arena and regional accreditation agencies, institutional surveys, and analyses 
of results from the VSA-conducted survey of participating institutions, and College Portrait database 
statistics and Google Analytics. 


Major Findings 


Our review found widespread agreement that the launch of the VSA in 2007 was a necessary response 
to the demands of the time. Accountability demands and public interests are transitory in nature and 
while the press for transparency persists, the environment in which the VSA exists is different from 
when it began. While seen as “necessary,” many observers also expect the VSA to continue to evolve. 


Even though participation by public universities is widespread, one third of eligible institutions do 
not participate in the College Portrait. Moreover, nearly half of participating institutions have not yet 
met the VSA stated expectations for the student learning outcomes pilot. One major concern is that 
the student learning outcomes section of the College Portrait attracts very few viewers. Several reasons 
may elucidate this shortfall, one of which is that the information posted and the manner in which it is 
presented do not reflect the needs and interests of users. 


We also found that the standardized tests of student learning originally approved for inclusion in the 
pilot lack credibility and acceptance within a broad sweep of te higher education community which, in 
turn, serves to undermine institutional participation in the VSA. Institutions participating in the VSA 
and other non-participating institutions would like to expand the number and nature of the student 
learning measures in order to more accurately portray student attainment and provide more useful and 
meaningful information for multiple audiences. 


Recommendations 


Given this mixed picture, we asked, “Should the VSA College Portrait be continued?” With some 
exceptions, the response was affirmative, suggesting “mend it, don’t end it” regarding the value of the 
VSA and the College Portrait. We recommend the VSA College Portrait be recast as a state-of-the- 
art electronic communication tool targeted for specific audiences, including prospective and current 
students, parents, and guidance counselors; alumni; faculty and staff; trustees; employers; accreditors; 
public ae makers; and media. 


We also urge that information presented on the College Portrait be presented around questions of 
particular interest to students and other relevant audiences to tell a contextualized, institution-specific, 
evidence-based story - possibly incorporating video or other media. To reach intended audiences, it 
will be necessary to design and implement strategies to draw traffic to the College Portrait website to 
increase viewership and access to information. A market-test of the next-generation College Portrait 
template would help confirm the language used is “consumer friendly” for respective audiences. 
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Executive Summary Continued 


Given the limited confidence and acceptance in available standardized tests as a means to provide 
evidence of student learning, we strongly recommend that the VSA College Portrait expand the range 
of accepted assessment tools and approaches. Such steps may also include an educative role to advance 
the development and utility of alternative methods of assessment. An additional dimension not yet 
explored is to create a space on the student learning outcomes section of the College Portrait where 
institutions can provide practical examples of how student learning evidence is being used to make deci- 
sions to improve student learning and enhance quality and performance. 


Finally, to expand viewership of the College Portrait, we recommend consideration be given to the 
creation of a College Portrait template that could be adapted and used by all postsecondary institutions, 
public and private, community colleges and others. A common template for all of higher education, 
while challenging to achieve, would serve as one access point for the public and thereby attract increased 
viewer traffic to the site. 


Conclusion 


The College Portrait student learning outcomes pilot was an effective response to the challenges 
emanating from the Spellings Commission and related pressures for accountability. However, it by 
no means will be sufficient in its current form going forward. The demand for evidence of student 
attainment will likely increase, along with the need to reach and better educate the public. Perhaps the 
single most important step APLU and AASCU can take is to transform the VSA from a compliance- 
focused vehicle to a platform through which institutions can speak to and engage their publics through 
evidence-based story telling. 
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Transparency & Accountability: 
An Evaluation of the VSA College Portait Pilot 


Introduction and Purpose 


The Voluntary System of Accountability, or VSA, was launched in 2007 by public four-year universities to supply 
comparable information about the undergraduate student experience through a common web reporting template 
— the College Portrait. The VSA is a partnership between the Association of Public and Land-grant Universities 
(APLU) and the American Association of State Colleges and Universities (AASCU). Originally supported by 
Lumina Foundation for Education, the VSA is now sustained through institutional membership dues ranging 
from $500 to $2500 based on institutional enrollment. 


The VSA has three primary objectives: 


1. To provide a mechanism for public institutions to demonstrate accountability and transparency, 
particularly in the areas of access, cost, student progress, and student outcomes. 


2. To provide a streamlined college search tool for students, families, and high school counselors by 
presenting clear, accessible, and comparable information on the undergraduate student experience 
through the College Portrait website (http://www.collegeportraits.org). 


3. To support institutions in the measurement of student learning outcomes through original research 
and by providing a forum for collaboration and exchange (http://www.voluntarysystem.org/index. 
cfm) 

The College Portrait was developed by a task force of university presidents, provosts, and other administrators. A 
particularly noteworthy feature of the College Portrait, and the focus of this evaluation, is the student learning 
outcomes page, conceived as a four-year pilot project for participating universities to gain experience in obtaining 
and publicly reporting information regarding student learning gains (value-added) at the institutional level using 
what are considered three comparable measures: Collegiate Assessment of Academic Proficiency (CAAP), Colle- 
giate Learning Assessment (CLA), and the ETS Proficiency Profile (ETS PP). Further information and background 
on the VSA, the College Portrait, and the student learning outcomes pilot project is located in Appendix A. 


In September 2011, VSA asked the National Institute for Learning Outcomes Assessment (NILOA) to evaluate 
the effectiveness of the student learning outcomes pilot project within the College Portrait. The NILOA effort 
was designed to capture available evidence to determine the extent to which the student learning outcomes pilot 
achieved the purposes articulated by its founders. While we tried to maintain a sharp focus on the student learning 
outcomes pilot, some focus group participants and personnel from eo organizations we interviewed 
shared broader views about the VSA initiative and the College Portrait in general. Believing that APLU and 
AASCU leadership would see these views as instructive, we have included them in our evaluation. The evaluation 
utilized a variety of data sources including the following: 


¢ Focus groups with over 150 APLU and AASCU members 

¢ — Interviews with 10 leaders from the policy arena and regional accreditation agencies 
¢ A survey of institutions that withdrew participation in the VSA 

¢ A survey of institutions eligible to participate in the VSA that decided not to join 

e Assurvey of guidance counselors and admissions personnel 

¢ An analysis of data from a VSA-conducted survey of participating institutions 

¢ An analysis of Google Analytics on College Portrait website traffic 

e An analysis of the College Portrait database statistics 


Appendix B contains additional information about the evaluation methodology including data collection and 
analysis. 
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Findings 


1. Both VSA-participating and nonparticipating institutions 
generally agree that the launch of the VSA in 2007 was 


a wise, timely, useful, and necessary response to the 
accountability and transparency demands of the time. 


Virtually every constituency viewed the VSA as a politically astute, proac- 
tive response to issues and concerns raised by policy voices at the time. The 
VSA College Portrait is credited with demonstrating a commitment to greater 
transparency of information and a willingness to respond to calls for account- 
ability through assessing and reporting learning outcomes. Those with whom 
we talked saw the VSA as a coordinated, collective response by higher educa- 
tion to an immediate public policy challenge. The VSA College Portrait was 
deemed to have encouraged more public reporting of information and to have 
drawn attention to the assessment of learning outcomes among America’s 
public universities. 


While there was general agreement that the VSA College Portrait was a neces- 
sary development, focus group participants and other observers agreed that 
it was not sufficient to quell public cynicism or to fulfill an immediate but 
constantly changing demand for accountability. Policy issues and public 
concerns are transitory, and the precise focus and nature of accountability 
interests and demands shift over time as events unfold and as actors and condi- 
tions change. The environment in which the VSA exists currently is different 
from when it began and while seen as a necessary response to the account- 
ability climate when created in 2007, many observers also expect the VSA to 
continue to evolve. 


Thus, while the VSA College Portrait, or something akin to it was and is 
important, those we talked to cautioned that it is far from sufficient and that 
a sustained effort is required to build transparency and earn the public trust. 


2. Many eligible institutions, about one third, do not 
participate in the College Portrait and nearly half of the 


participants have not yet met expectations set forth in 
the College Portrait student learning outcomes pilot. 


Six out of ten (62%) of eligible APLU and AASCU institutions presently 
participate in the VSA College Portrait.’ As of February 2012, 320 institutions 
were art oak or had participated in the VSA, while another 171 institu- 
tions eligible to participate had not done so. Most institutions joined the VSA 
initiative in 2008 (Figure 1), with 75% of currently participating institutions 
signing on that year. Since 2007, 25 institutions have withdrawn from the 


VSA. 


' In the NILOA survey of non-participating institutions, 8% stated they had not been aware 
of the VSA and were now considering joining. 


the VSA is a voluntary initiative 
which provides research, support, 
and an online template called the 
College Portrait for participating 
institutions to use to assemble 
and disseminate information 
and demonstrate accountability. 
http://www.voluntarysystem.org/ 
index.cfm 


The VSA’s College Portrait is 

a website created to provide 
readily accessible transparent, 
comparable information. 
http://www.collegeportraits. 
org/ Relevant information from 
each participating institution is 
housed on the College Portrait 
website. This information 
includes results from the student 
learning outcomes assessment 


pilot. 


In this report, “VSA” refers to 
the larger project that hosts 
the College Portrait website. 
“Individual institutional 
College Portraits” refers to the 
institution-specific College 
Portraits available on the VSA 
College Portrait website. 
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Figure 1. Number of Institutions Joining VSA by Year of Membership 


Of VSA participating institutions, substantial numbers do not publish a 
College Portrait online each year: 


* 45 institutions (16%) did not publish a College Portrait in 2008 
* 27 institutions (9%) did not publish a College Portrait in 2009 

¢ 54 institutions (17%) did not publish a College Portrait in 2010 
* 58 institutions (18%) did not publish a College Portrait in 2011 


More than half (55%) of participating institutions that do publish their College 


Portrait have yet to post student learning outcomes information (Figure 2). 


The College Portrait student learning outcomes pilot allows institutions up 
to four years to report the results from one of the three approved value-added 
measures administered to first-time first-year students and then to senior 
students. Thus, results from the majority of VSA institutions, which joined in 
2008, should be posted by December 31, 2012 at the conclusion of the pilot. 
In addition, analysis of the results of a survey of VSA-participating institutions 
indicates the majority of institutions that have yet to post results have—in 
fact— administered one of the three approved tests and are now waiting for 
the results; in some instances, schools have only tested first-year students or 
seniors and are waiting to test the second group. 


Among the 144 institutions that have posted standardized test results on their 
institutional College Portrait, 86% administered the CLA, 9% administered 
the ETS PP, and about 5% administered the CAAP. 


“We are still in the process of 
assembling the data required for 
the Portrait. The first time we 
administered the test we did not 
have enough participants - we 
have since given it a second time 
and have not yet received the 
results. We intend to post after an 
internal review of the results once 
received.” 


Focus Group Participant 
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Figure 2. Number of Institutions Posting Standardized Test Results on the 
College Portrait 


In addition to posting the value-added standardized test results on the student 
learning outcomes page, the VSA requires institutions to include a link to 
institution-specific learning outcomes data and provides space for a narrative 
description of assessment activities on the College Portrait. In NILOA’s review 
of individual College Portrait sites, only two thirds of participating institu- 
tions provided the required links, and several of these were inoperative. Links 
that were working sent viewers in a variety of directions, some to program- 
level assessment activities, others to summaries of NSSE or CLA results, or 
to pages describing institutional research or assessment office information. 
The narrative information of institutional assessment activities appeared on 
roughly two thirds of participating institutions’ College Portraits and varied 
in length and detail. 


By and large, the narratives of learning outcomes information, including the 
interpretations of test scores, were suitable for internal audiences familiar with 
the lexicon of student learning outcomes assessment and continuous improve- 
ment. 


As a result, College Portrait viewers not directly involved with or knowledge- 
able about assessment themselves would likely be hard-pressed to comprehend 
the provided information and understand its implications for student learning. 
Overall, only a minimal amount of information was posted on institutions’ 
College Portrait student learning outcomes pages. What did appear typically 
lacked helpful explanatory or contextual material that would aid the viewer in 
understanding a the results of the standardized tests meant or in deriving 
meaning from the display. 


3. The student learning outcomes section of institutional 


College Portraits attracts little traffic. 


Considerable investment of time and resources is required by an institution to 
gather and enter the information for the College Portrait including adminis- 
tering the student learning outcomes test, placing these data on the website, 
and contextualizing and communicating results. For this reason it is impor- 


“We clearly have much more 
data than we ever had before 
and we do a lot more with it - 
but numbers have no meaning 
unless you provide context and 
show what this means over 
time, including across multiple 
measures, to provide a bigger, 
meaningful picture.” 


Focus Group Participant 
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tant to know whether potential users are viewing the information presented | ee 


there—especially, given the purposes of this evaluation, the student learning 
outcomes section of the template. In both the focus groups with institutional 


VSA participants and in conversations with public officials, concern was “It is difficult to evaluate 
expressed that too few people know that the student learning outcomes site the ‘accountability value’ of 
exists. CO inden eee 

participating in the VSA because 
Google Analytics provides information about website traffic in terms of the as an institution, we have no 
number of visitors to specific pages as well as the average time users spend way of knowing who accesses the 


on a page. NILOA obtained access to the College Portrait Google Analytics 
from the VSA, and using time spent on a page as a proxy for interest, website 
traffic was examined across the entire College Portrait site as well as for indi- Survey Respondent 
vidual College Portraits from October 2009 through August 2010 and from 

February 2011 to December 2011. 


College Portrait.” 


‘The analytics from the 2011 time period for the entire College Portrait site 
showed that very few visitors looked at an institutional student learning 
outcomes page; many more users viewed college cost information (Figure 3). 
Mote specifically, of the 1,157,520 unique page views of the College Portrait 
site as a whole, 43% were of the College Cost Estimator, while only 1% of the 
entire page views (a total of 15,862 unique pages views) were of the student 
learning outcomes page. 


College Cost Estimator 
College Cost & Financial Aid 
Students 

Admissions 

Student Success & Progress 
Classes & Campus Life 
Majors, Graduation, & Next 
Student Learning Outcomes 


Student Experiences 


0 5 10 15 20 25 30 35 AO 45 
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Figure 3. Percentage of Total Views of the College Portrait Website by Information 
Category 


Both the number of views of the student learning outcomes page of individual 
College Portraits of the 320 participating institutions and the time spent on 
the learning outcomes page were low. About 82% of participating institutions 
had traffic on their student learning outcomes page in 2009-2010 and 2011; 
however, almost one in five of participating institutions did not have a single 
visitor to their student learning outcomes page during either time period. People 
from several of the participating institutions mentioned in focus groups that 
they review their individual College Portrait analytics or ask at orientation or 
during admissions meetings if anyone viewed the College Portrait site. In most 
cases, few hands are ried. 
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‘The length of time users spent on the individual College Portrait student 
learning outcomes page ranged from a 17-minute visit at one institution's site 
to several sites where users spent no time at all. In 2011, the longest amount 
of time spent on a student learning outcomes page was roughly four minutes. 
Figure 4 shows the range of time spent on the student learning outcomes page. 
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Figure 4. Number of Views of the College Portrait Student Learning Outcomes 


Page by Length of Time Spent on Page by Year 


The number of people who visited an individual College Portrait site and then 
viewed the student learning outcomes page was minimal, with a slight decrease 
in 2011. Of the total number of users who came to an institution’s College 
Portrait in 2009-2010, only 9% examined the student learning outcomes 
page for an average of 2 minutes and 17 seconds. Only 7% of users who came 
to an institution’s College Portrait site in 2011 examined the student learning 
outcomes page for 23 seconds—substantially less time than the year prior. 


Institutions receiving the highest number of views on the student learning 
outcomes page tended to link their institutional College Portrait to their home 
or admissions page, which is the link required in the participation agreement 
of the VSA. The agreement states that participating institutions will include 
the College Portrait logo on the institution’s home page or another appropriate 
page that is viewed frequently by prospective students and not more than two 
clicks away from the institution’s home page (http://www.voluntarysystem. 
org/participants/signup.cfm). Another NILOA study (Jankowski & Provezis, 
2011) found that three quarters of VSA-participating institutions mentioned 
their participation in the College Portrait somewhere on their institutional 
website; however, the majority of these posted the information on their insti- 
tutional research or assessment page, while only 9% referred to the VSA on 
their home or admissions page. 


Some campuses include information on their College Portrait that goes 
beyond basic VSA requirements. The California State University System, for 
example, displays a California Public Good page. Since that page has been 
available, 10% of users who view one of the participating California institu- 
tions’ College Portraits also visit the Public Good page and view it for an 
average of one minute. 


“We have not talked about 

the College Portrait and the 
student learning outcomes pilot 
specifically since we joined - it 
has moved to the back burner 
and has not been a discussion 
item on our campus or in our 
system.” 


Focus Group Participant 
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The relatively low level of viewer traffic to the College Portrait information 
prompted some respondents to ponder whether the amount of time required 
to gather the necessary data and populate the College Portrait was worth the 
effort. At some institutions much of the information about student learning 
already appears on institutional websites or is contained in reports to state 
agencies. In addition, many respondents were concerned that the student 
learning outcomes information required for the College Portrait did not 
adequately represent the actual assessment work underway at institutions. 
Some respondents suggested that if the College Portrait could be populated 
by the VSA drawing information from other reporting requirements and insti- 
tutional websites, it would reduce redundancy, lower costs, and not unduly 
burden institutions. 


Focus group participants wondered whether the College Portrait could serve 
a clearinghouse function whereby multiple sources of information could be 
consolidated in a user-friendly format that appeals to diverse audiences. Several 
respondents suggested that the VSA, as opposed to institutions, consolidate or 
“pull” relevant information from state-required accountability reports. Others 
suggested the VSA consider creating a report on higher education from the 
College Portrait information utilizing peer comparison, similar in form to the 
Wisconsin accountability reports including comparison to selected peers (see 
http://www.wisconsin.edu/opar/accountability/). The University of Wisconsin 
System accountability reports also suggest the potential for aligning VSA more 
closely to state reporting requirements or for making the VSA a repository for 
state accountability data. 


In short, traffic to the College Portrait and the learning outcomes section has 
been distressingly low. Addressing that reality presents a major challenge to the 
VSA going forward. 


. The information posted on the College Portrait may not 
reflect the needs of prospective students and families 


or provide the information they actually seek to make 
decisions about where to attend college. 


Many participating and nonparticipating institutions as well as others with 
whom NILOA spoke shared the view that the student learning outcomes 
information on the College Portrait is not user friendly. In large part, this is 
because the learning outcomes page of many participating institutions is blank 
or contains very little information. As noted earlier, the information is typi- 
cally couched in language familiar to insiders rather than pitched to the needs 
and interests of prospective students and the public. Results of the standard- 
ized tests that are posted on the College Portrait, for example, are not often 
explained in layperson terms or contextualized in a meaningful manner. 


Institutions report results of student learning so that prospective students and 
their families will consider the information when de ing where to go to 
college and, consequently, make better-informed decisions. This is because 
it is thought policy makers, parents, and prospective students want simple, 
comparable indicators. “Simple” quickly becomes complex, however, given 
the diversity of campuses, missions, and programs, as well as the varied needs 
of students and the public. Student college choice experts agree that more 


“We report on measures to our 
board, to state agencies, and 

to others. We wish there were 
ways that VSA could download 
available data and then ask 
institutions for data they could 
not get from other available 
sources to lower cost and 
redundancy for institutions.” 


Focus Group Participant 
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information is needed to help students make informed college choices (Long, 
2007; Manski & Wise, 1983) due to the variety of factors shaping these deci- 
sions such as geographic location, institutional fit, campus setting, distance 
from home, and cost (Ewell, 2009; Kinzie, Palmer, Hayek, Hossler, Jacob, 
& Cummings, 2004; McCormick, 2010; McDonough, 1997). On the other 
hand, the assumption that information must be comparable across similar 
institutions to facilitate student choice may or may not have advanced compa- 
rability, but it clearly appears to have limited the richness of information that 
might have been of potential use to students and the public. 


Achieving public understanding of test scores on the three different VSA- 
approved measures is a major challenge. While the VSA validity study showed 
that the three tests measure similar constructs (Klein, Liu, & Sconing, 2009), 
it is less clear how results from these tests can be used by students to inform 
college choices. Because we know so little about what students and parents 
believe they need to know regarding student learning when selecting a college, 
it is difficult to know what evidence to include on the College Portrait that 
would respond to their needs. What is clear, as noted earlier, is that few people 
are looking at this information as currently presented. 


. The authorized standardized test measures of student 
learning outcomes lack broad credibility and acceptance 
in the higher education community, undermining 


institutional participation and engagement with the 
VSA and campus faculty and staff support of the VSA 
initiative. 


It is commonly understood in assessment circles that no single test or measure 
can capture the full range of student learning or the impact of college on 
other aspects of the student experience. To address this issue at the outset, 
VSA leaders evaluated 16 potential learning outcomes tests and recommended 
three as options for use in the VSA (Shulenburger & Keller, 2009), offering 
institutions the opportunity to choose from among the three instruments the 
one best suited to their particular campus. As noted earlier, the three approved 
instruments are the Collegiate Learning Assessment (CLA), the Collegiate 
Assessment of Academic Proficiency (CAAP), and the ETS Proficiency Profile 
(formerly MAPP). 


In a survey of participating institutions including those that have yet to post 
results on their College Portrait, 82% reported having administered one of the 
three tests, with the CLA and ETS PP being the most popular choices, at 41% 
and 39% respectively. However, as noted earlier, the majority of institutions 
have not yet reported results, with fewer than half (45%) listing results on 
their College Portraits, and with most of those being CLA results. 


Survey respondents and focus group participants raised numerous concerns 
about displaying the results of standardized tests. The primary criticism was 
the inability of campus personnel to understand what the test scores repre- 
sent. Critics questioned precisely what the tests measure—initial ability of 
incoming students or learning gains as a result of the college experience. In 
addition, many with whom we spoke lacked confidence in the reliability and 


“We should not be using 
nationally-normed standardized 
tests. They have many serious 
drawbacks as measures of actual 
student outcomes. Only 10% of 
students self-reported that they 
gave the test their best effort. 

It makes me wonder just how 
worthwhile it would be for 
comparison purposes with other 
institutions.” 


Survey Respondent 
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validity of the results, in interpreting the scores, and in using the test results to 
improve student learning. This is because the information from the tests was 
not sufficiently disaggregated to identify what part of the course sequence or 
curriculum needed revision or improvement. Most respondents on this topic 
said that program and course embedded assessments were better approaches 
than the standardized tests. One respondent described particularly well what 
many expressed: 


We see no value in administering any of these tests other 
than for VSA compliance. They are costly and do not give us 
reliable information that can be used to either represent or 
improve learning at the institution. We would prefer to have 
VSA require a balanced and accountable use of these sorts of 
methods [a variety of approaches] rather than requiring one 
of the three big tests. 


Survey respondents and focus group participants expressed concern about 
campus-to-campus variations in the procedures used to select student partici- 
pants and to administer the test. Moreover, in many instances the tests are 
completed by numbers of students too small to draw conclusions and point 
to steps institutions can take to improve student learning. In addition, the 
motivation of the students and the time they spend on the test (Hosch, 2010) 
may influence the test results. The VSA survey of participating College Portrait 
institutions found that many of the institutions had difficulty acquiring a 
representative sample of enough students to sit for the test. These and ee 
concerns raise questions about data quality. One respondent spoke for many: 


We faced considerable challenges in getting a reasonable 
sample of students to respond and doubted, based on our 
observations, that students performed to the best of their 
ability on the test. We are not completely satisfied that the 
test is a satisfactory measure of student learning, or that 
testing conditions can be managed in ways that will yield 
meaningful results. We don’t know what the results represent, 
but we do not believe they are representative of learning gains 
by undergraduates at our institution. 


In a survey of institutions that withdrew from the VSA, similar concerns were 
raised with 40% stating they withdrew because of the standardized test require- 
ments. 


¢ Four fifths (81%) of survey respondents said that the student learning 
outcomes pilot requirements did not align with their assessment 
aaa at the program level and that the test results were not usable 

or campus improvement efforts. 


¢ A third (35%) of the nonparticipating institutions reported they 
would join if some flexibility were introduced to the reporting of 
student learning outcomes. 


¢ 11% of institutions administer the standardized tests for alternative 
purposes, such as a rising junior-year assessment tool. 


e A quarter of institutions (24%) frequently expressed concerns and 
fears over the potential misuse of the results. As one respondent put 
it, “We do not think it is appropriate to list numbers without explana- 
tion of results.” 


“We have successfully crafted 
several assessment instruments 
that are tailored to our student 
learning objectives. We are 

not interested in adopting an 
external testing mandate that 

is not in alignment with our 
student learning outcomes and 
established assessment practice.” 


Survey Respondent 
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All this said, despite widespread concern and angst among institutions, the 
sentiments of most with whom we spoke indicated that current participants in 
the VSA College Portrait may be inclined to continue to administer one of the 
tests as long as they are required. 


6. Participating institutions generally agree that expanding 
the number and nature of the student learning measures 
will produce more accurate portrayals of student 


attainment and more useful information for campuses, 
and will make the information more meaningful for all 
audiences. 


In late 2011, the VSA board voted to expand the range of student learning 
outcomes assessment options. Several potential items for expansion of the 
learning outcomes section of the College Portrait were offered by focus group 
participants and survey respondents. For each of the potential items, respon- 
dents stated that they wanted “to tell our student learning outcomes story 
with measures appropriate to us”—an issue particularly important for institu- 
tions with large numbers of transfer students. Universities wanted to ensure 
that their distinctiveness could be expressed through other reporting options 
including authentic measures of learning, and that student performance at the 
program level could be represented. Some items for consideration mentioned 
by respondents include the following: 


¢ Association of American Colleges & Universities (AAC&U) VALUE 
rubrics 


e Alumni surveys 

¢ Licensure pass rates 

¢ Accreditation information 
¢ Program-level assessment 
¢ Portfolios 


The VSA survey of participating institutions offered other insights into the 
types of assessment approaches and tools institutions consider useful (Figure 


5): 


¢ The most commonly mentioned were the AAC&U VALUE rubrics 
(38%) and the ETS Major Field Achievement tests (42%). Respon- 


dents deemed both of these useful to their institutions. 


¢ Among survey respondents from institutions eligible for but not 
participating in the VSA, 35% indicated that the addition of the 
AAC&U VALUE rubrics as an option to report student learning 
outcomes would positively influence their decision to join the VSA. 


“There is a political 
accountability issue where 
course-level assessment that 
we do does not meet the 
public accountability we are 
talking about. The course- 
level assessment does not 
provide the value-added for 
the public that they need for 


accountability, but it should.” 


Focus Group Participant 
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Percentage of Respondents 


Use Useful 


Figure 5. Percentage of Respondents Claiming to Use Assessment Approaches and 


the Percentage That Find Them ‘Useful’ 


Another theme that emerged was the suggestion to broaden the College 
Portrait to include program-level information and to disaggregate results 
according to different types of students such as transfer, associate, and bacca- 
laureate degree students (23%). The use of student learning outcomes infor- 
mation by the public and policy makers presents special challenges—including 
that of helping various publics distinguish between student learning outcomes 
evidence and other performance indicators such as graduation rates, retention, 
and job placement statistics. All are relevant and yet, at times, confusing. One 
survey respondent expressed the idea this way: 


It’s important to educate outside audiences about the variety 
of appropriate methods for evaluating student learning 
outcomes instead of relying on external tests that provide 
little actionable feedback and discredit the expertise of our 
faculty. Instead, the VSA could give us a way to highlight 
the value our faculty bring to evaluating the work of their 
students. 


Some respondents also suggested opening up participation in the VSA to all 
public and private institutions. Because there are multiple templates in use 
now, the industry and our various constituents could be better served with 
a single template. As one interviewee stated, “For all the good the VSA has 
accomplished, the level of institutional participation has not reached the 
needed scale to address public cynicism.” Expanding membership as well as 
the range of measures conveys a sense that a broader focus on student learning 
would address the deeply rooted and widely shared concern that the stan- 
dardized tests do not accurately reflect learning that occurs at the program 
and course levels and would allow for the presentation of a more compre- 
hensible picture to the public. Such an approach could be an effective means 
for various stakeholders to learn information about higher education institu- 
tions—perhaps in collaboration with other projects like the Delta Cost Project 
(now administered by American Institutes for Research)—and state reporting 
requirements. 


“It has been a pilot program 
and whoever gets it right in the 
first couple years? We need to 
answer the question: What is 
the long-term outcome we want 
to see?” 


Focus Group Participant 
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Conclusion and Recommendations 


Public universities understand and support the value of the VSA College 
Portrait and view it as a creative, constructive, politically astute response to 
the accountability pressures at the time it was developed. “Should the VSA 
College Portrait be sustained,” we asked? With some exceptions, the response 
was a resounding “yes,” suggesting a “mend it, don’t end it” view from the 
field. Even though the College Portrait student learning outcomes pilot seems 
to have been an effective response to the challenges emanating from the Spell- 
ings Commission, by no means will it be sufficient going forward. As noted 
earlier, while the interests and pressures of government and policy makers shift 
over time, the need for evidence of student attainment is not likely to abate. 
To be sustainable and to weather changing interests, the VSA could transform 
from a compliance-focused vehicle to a platform through which institutions 
can speak to and engage their publics through evidence-based story telling. 


The development and implementation of the VSA College Portrait was 
prompted by external forces. If the VSA College Portrait is to be “mended” and 
sustained, APLU and AASCU would be well advised to find ways to modify 
the purposes, foci, and use of the College Portrait to make it more congruent 
with the actual needs and interests of the multiple audiences it is meant to 
serve. The College Portrait must become a medium through which partici- 
pating institutions organize and share information in meaningful ways—ways 
that are responsive to the legitimate interests and concerns of various audiences 
including students and parents/families, the media, policy makers, trustees, 
employers, and other higher education stakeholders. 


Perhaps the single most important step APLU and AASCU can take is to transform 
the College Portrait from a compliance-focused vehicle to a platform through which 
institutions can speak to and engage their publics through evidence-based story 
telling. Public trust in most societal institutions is at a low ebb. There are indi- 
cations that the public’s appetite for accountability is more about gaining trust 
in what we say to them than it is about the statistics we offer. Coupled with 
this is skepticism within the academy about using test scores as the primary 
vehicle to build public trust. Numbers standing alone out of context can be 


manipulated in ways that obfuscate rather than illuminate (Johnson, Roch- 
kind, & DuPont, 2011). 


Toward this end, we offer the following recommendations. As noted earlier, 
our comments extend in some instances beyond the student learning outcomes 
pilot effort to reflect the broader views of those with whom we talked during 
the course of this evaluation. 


1. Recast the VSA College Portrait as an effective communication tool using 
state-of-the-art design technologies targeting specific audiences. The 
relevant information should be organized around the documented 
interests and questions of specific audiences. This will likely require a 
template with portals tailored to specific audiences: prospective and 
current students, parents/families, and guidance counselors; alumni; 
faculty and staff; trustees; employers; accreditors, public policy 
makers, and the media. In its current form, the meaning and implica- 
tions of the information on the template are not self-evident, espe- 
cially for those outside the academy. It is difficult to imagine a single 
display or page that can adequately support the information needs 
and interests of all audiences. Focusing on specific audiences, framing 
questions carefully, and presenting information meaningfully and in 
context should be the prime challenge of the next version of the VSA 
College Portrait. 


APLU and AASCU would be well 
advised to find ways to modify 
the purposes, foci, and use of the 
College Portrait to make it more 
congruent with the actual needs 
and interests of multiple audi- 
ences... 
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Contextualize the information presented in the College Portrait and— 
where possible—frame the information around questions of interest to 
the specific intended audience. Ideally, the information will be assem- 
bled and interpreted to tell an evidence-based story, possibly aided 
by video or other media. The next version of the College Portrait 
template needs to be market-tested to be sure it uses consumer-friendly 
language for the respective audiences. For some audiences, even the 
phrase “student learning outcomes” may be puzzling or off-putting. 
Perhaps other College Portrait pages can include prompts encour- 
aging viewers to visit the student learning outcomes page. Further, 
information regarding academic quality and performance should be 
available throughout the College Portrait, especially in areas of higher 
traffic. Whatever the ultimate solution, it is not helpful to campuses 
or the larger public to populate and maintain such sites if the target 
audiences routinely ignore them. 


Expand the range of assessment tools and cs ss institutions can use 
to provide evidence of student learning and to serve an educational role 
in advocacy for alternative methods of assessment. One goal of the VSA 
was to encourage universities to measure and report student learning 
outcomes using local institutional research. The VSA Board’s decision 
to ease the required administration of a single standardized test and 
posting of results on the College Portrait is a step in the right direction. 
Not only has this provision discouraged institutional participation in 
the VSA, it is not clear that the test data have provided tangible benefit 
to the intended publics. Equally important, restricting the reporting 
of student learning outcomes to a test score may have led campuses 
to ignore the many other relevant indicators of student learning that 
might have been shared ‘The design of the next version of the College 
Portrait template should serve an educational and advocacy role for 
alternative assessment methods that use authentic student work to 
make judgments about the quality of student learning. 


Design and implement strategies to draw traffic to the College Portrait 
website to increase viewership and access to information. Where infor- 
mation is featured on websites makes a difference. In addition to 
participating institutions posting the College Portrait logo on their 
institution website, the VSA could broaden marketing efforts to raise 
general awareness of and bring traffic to the College Portrait site. 
According to the VSA participation agreement: 


The institution will include the VSA/College Portrait logo on the 
institution’s home page or another appropriate page (e.g., admis- 
sions page) that is regularly visited by prospective students and 
not more than one or two clicks off the institution’s home page. 
An active link will be maintained from the logo directly to its 
College Portrait http://www.voluntarysystem.org/docs/SignUp/ 
VSAParticipationAgreement.pdf. 


Because a substantial proportion of participating universities do not 
adhere to this agreement, we urge the VSA Oversight Board to ponder 
how this sensible criterion for membership can be faithfully followed 
by member institutions. 


Create a space in the student learning outcomes section of the College 
Portrait where institutions show examples of how student learning 
evidence is being used to make decisions and to improve student learning 
and institutional performance. Such examples will not only increase 
the awareness of faculty and staff of the possibility and desirability 
of data-informed improvement, but they will also demonstrate to 


The next version of the 
College Portrait template 
needs to be market-tested 
to be sure it uses consumer- 
friendly language for the 


respective audiences. 


The design of the next version 
of the College Portrait 
template should serve an 
educational and advocacy 
role for alternative assessment 
methods that use authentic 
student work to make 
judgments about the quality 
of student learning. 
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external audiences that public universities are not complacent about 
or satisfied with the status quo and are taking concrete steps to 
improve. 


Consider developing a College Portrait template that can be adapted 
and used by all postsecondary institutions. It is difficult enough for 
the public—especially those who have never worked in a college 
or university—to understand and appreciate the learning outcomes 
campuses expect students to attain. When this information is 
presented in multiple formats using different terms for the same 
concepts, no wonder few users take the time to master the complexi- 
ties. Expanding institutional membership and participation in the 
VSA College Portrait will both serve the aims of the VSA and the 
greater public interest as well as increase traffic to and use of the site. 
Ideally, every public university would participate in the VSA College 
Portrait. Inviting participation by non-APLU/AASCU members such 
as private colleges and universities and community colleges should 
be explored as well. The more widely used the College Portrait, or 
some comparable format, the more accessible it will become to the 
general public and target audiences. Competing formats and brand 
names are not helpful to the public or to the academy. Achieving 
consensus around a common format of information may be beyond 
the reach of the higher education community. Still, if we are to reach 
the American public, coming together around a common template 
would greatly expand access and improve communication. 


“Student learning and how 
to show it is not an irrelevant 
thing to be working on and 
thinking through, and it is 
not easy. But, we have to get 
this right.” 


Focus Group Participant 
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Appendix A: 


Voluntary System of Accountability: An Overview 


Calls for accountability and improved student learning in recent years have challenged colleges and universities to be 
more transparent in terms of student and institutional performance. These calls have become more insistent as access 
to higher education has increased, state support has declined, the national economy has faltered, and graduation rates 
have remained unacceptably low (Borden & Young, 2008; Commission on the Future of Higher Education, 2006; 
McPherson & Shulenburger, 2006a, 2006b; Provezis, 2010). Taken together, these circumstances have made efforts to 
be transparent about institutional performance and to compare the performance of institutions even more important. 
One initiative that has emerged in response to these concerns is the Voluntary System of Accountability, or the VSA.! 


The VSA was founded in 2007 in response to the deliberations of the 2006 Spellings Commission on the Future of 
Higher Education and the looming 2008 Reauthorization of the Higher Education Act. Originally supported by Lumina 
Foundation for Education and now sustained through institutional membership dues ranging from $500 to $2500 based 
on institutional enrollment, the VSA is a voluntary initiative for public four-year colleges and universities that provides 
information to students, families, policymakers, campus faculty and staff, the general public, and other higher education 
stakeholders. The information about student learning and institutional performance was intended to (a) facilitate college 
student choice; (b) inform institutional decision-making for improvement; and (c) respond to accountability pee 
The VSA evolved through a partnership between the Association of Public and Land-grant Universities (APLU) and the 
American Association of State Colleges and Universities (AASCU) (McPherson & Shulenburger, 2006a, 2006c; Miller, 
2008). Oversight for the VSA is provided by a VSA Oversight Board made up of senior officials from APLU and AASCU 


institutions. According to its website, http://www.voluntarysystem.org/index.cfm, the VSA has three primary objectives: 


1. To provide a mechanism for public institutions to demonstrate accountability and transparency, particularly in 
the areas of access, cost, student progress, and student outcomes. 


2. To provide a streamlined college search tool for students, families, and high school counselors by presenting 
clear, accessible, and comparable information on the undergraduate student experience through the College 
Portrait website: www.collegeportraits.org. 


3. ‘To support institutions in the measurement of student learning outcomes through original research and by 
providing a forum for collaboration and exchange. 


To serve these objectives, a task force of university presidents, provosts, and other administrators helped develop the 
College Portrait—a common web-reporting template that communicates information on the undergraduate student 
experience. The College Portrait is designed to 1) demonstrate accountability and stewardship to the public; 2) measure 
educational outcomes to identify effective practices; and 3) assemble information that is accessible, understandable, 
and comparable (http://www.collegeportraits.org/about). 


The College Portrait is composed of a variety of informational pages including: 


¢  Students—Information on the characteristics of students who attend 

¢ Admissions—Information on the number of students who applied, were accepted, and enrolled 
* College Costs & Financial Aid—Information on cost and financial aid 

* College Cost Estimator—A cost calculator 

* Classes & Campus Life—Information on professors, student housing and campus safety 

¢ Student Experiences—Information on NSSE results 

* Majors, Graduation & Next Steps—Information about majors and graduates 

* Student Success & Progress—Information on degree completion and time to degree 


° Student Learning Outcomes—Information on what learning gains to expect in critical thinking, writing, and 
other important subjects 


' The VSA was not the only response, other web reporting templates include Transparency by Design (TbD), University and College 
Accountability Network (U-CAN), and the Voluntary Framework of Accountability (VFA). 
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When the College Portrait was developed, it was determined that the student learning outcomes information presented 
would take the a of a pilot project to be reevaluated in 2012. Access to the student learning outcomes assessment 
information required in the pilot project is located on the right-hand side of individual College Portraits, at the bottom 
of the list of additional options (see image from the website below). 


Sep Mevgation © Admin Login 


Sample University at Sample Town 


Sample University 
123 College Lane 7, College Portrait 
College Town, ST 12345 
ETD compare Print Portrait Contact 
Sampce University Learn More 
Students 4 


This sample University is among the top public universities in the Midwest, 

preparing students to excel now and the future. Founded in 1888, Sample Admissions > 
U is a land grant college founded from the first Morrill Land Grant Act. Stay- 

Ing true to its mission, our institution with its distinct academic programs 


serves its students and community. The university is proud of its values and College Costs & Financial Aid : 

long term commitments. Sample University helps people appreciate life and 

learning and contribute meaningfully to the lives of others. College Cost Estimator 4 
Classes and Campus Life ? 


Student Experiences » 


sl ee Gait scat 
— Majors, Graduation and Next Steps * 


ee 
Student Learning Outcomes J 
es es es 


If desired, institutions may also include a clearly labeled supplemental page located after the student learning outcomes 
page option on the College Portrait. Several institutions have utilized this opportunity, including the California State 
University (CSU) system with its inclusion of a section on public good—a common CSU page where CSU campuses 
communicate their value to the public. For example, see CSU’s postings at http://www.collegeportraits.org/CA/CSU- 
Bakersfield/content_page and. http://www.calstate.edu/fyi/2011/FYI_May2011.pdf or the College of New Jersey’s 
consumer information report option at http://www.collegeportraits.org/NJ/TCNJ/content_page. 


Since its inception, the College Portrait website has undergone several changes including the addition of a college cost 
calculator and an overall new visual design in December 2011. The revised College Portrait website makes it possible to 
compare institutions side by side on a variety of quality indicators and cost. However, student learning outcomes assess- 
ment information is not part of the comparison information at this time. 


Participation Agreement 


As part of the process of becoming a member of the VSA, institutions sign a participation agreement stipulating they 
will follow the reporting requirements, conditions, and timelines set by the VSA (http://www.voluntarysystem.org/docs/ 
SignUp/VSAParticipationAgreement.pdf). Issues addressed in the agreement include VSA membership dues, provi- 
sions regarding withdrawal trom the VSA, and conditions for maintaining the accuracy of data reported on the College 
Portrait website. The agreement also specifically outlines what is expected of member institutions during the student 
learning outcomes pilot project. These requirements, found in the Measurement and Reporting of Student Learning 
Outcomes section of the agreement, state that institutions must 


16. Link to institution-specific learning outcomes data within three months. The data reported may include, but is 
not limited to, reports on program assessments, employer satisfaction with graduates, licensing exam pass rates, 
etc. 


17. Participate in the VSA pilot project to measure student learning outcomes by selecting one of three tests to 
measure student learning gains: 
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a. Collegiate Assessment of Academic Proficiency (CAAP)—two modules: critical thinking and 


writing essay; 


b. Collegiate Learning Assessment (CLA)—including performance task, analytic writing task; 


ETS Proficiency Profile (formerly known as MAPP)—two subscores of the test: critical 
thinking and written communication; either the standard or the abbreviated form can be used. 


18. Administer the selected test to random samples of first-time freshmen and seniors who entered the institution 

as freshmen (based on their student population, institutions may alternatively elect to measure student learning 

ains for incoming transfer students and seniors who entered as transfers). ‘The institution will follow the guide- 
fines of the test maker in selecting the sample. 


19. Report the results of the pilot project on the College Portrait no more than four years after becoming a VSA 
participant and update results at least every three years. 


20. Provide a link to information on the test administration, sample, and response rate. 


Validity Studies 


An important activity undertaken by the VSA, the Test Validity Study (2009), was to determine the degree to which the 
three student learning measurement tools approved for use in the College Portrait measured, at the institutional level, 
similar proficiencies related to broad cognitive skills (critical thinking, analytic reasoning, and written communication). 
‘The three approved instruments are the Collegiate Learning Assessment (CLA), the Collegiate Assessment of Academic 
Proficiency (CAAP), and the ETS Proficiency Profile (formerly the MAPP). The Test Validity Study examined if these 
three tools measured the same cognitive abilities through the value-added approach. The value-added approach was 
based on the Council for Aid to Education (CAE) administrations of the CLA instrument. Value-added scores (student 
learning gains) are calculated as the difference between the actual and the expected scores of graduating and entering 
students after controlling for academic ability. Even following the Test Validity Study and several other studies of the 
standardized tests, concern has remained over the validity of the tests as an accurate measure of student learning and as 
a measure of institutional effectiveness.” Further, there have been concerns that the tests are not comparable and that the 
results presented from them are difficult to understand—thus further obscuring differences between institutions (Kelly 
& Adelman, 2010). 


> For additional information see the Test Validity Study report (2009), the Rand study on the CLA (Hardison & Vilamovaska, 2009), the study 
by the Education Sector and the American Enterprise Institute (Kelly & Adelman , 2010), and the critique by McCollum (2011). Of potential 
interest on the use of results are publications of the Council of Independent Colleges’ (CIC) consortium on the CLA involving liberal arts 
institutions, which provide information on institutions that used the results to improve student learning (CIC, 2008; Paris, 2011). 
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Appendix B: Methodology 


The National Institute for Learning Outcomes Assessment (NILOA) conducted an evaluation of the student learning 
outcomes pilot project of the College Portrait. This evaluation, to determine the efficacy of the student learning outcomes 
pilot project, gathered information and feedback from institutional leaders and various higher education stakeholders 
about the overall effectiveness and relevance of the pilot. The evaluation also reviewed and analyzed existing literature 
and College Portraits, institutional websites, and data obtained from previous NILOA studies. The evaluation utilized a 
variety of data sources including 


¢ Focus groups with over 150 APLU and AASCU members 

¢ — Interviews with 10 leaders from the policy arena and regional accreditation agencies 
¢ A survey of institutions that withdrew participation in the VSA 

¢ A survey of institutions eligible to participate in the VSA that decided not to join 

e Asurvey of guidance counselors and admissions personnel 

e An analysis of data from a VSA-conducted survey of participating institutions 

¢ An analysis of Google Analytics on College Portrait website traffic 

¢ An analysis of the College Portrait database statistics 


The evaluation of the student learning outcomes pilot project for the College Portrait sought to address the following 
main question, with several related subquestions: 


¢ How effective and relevant is the VSA approach in supporting the measurement and reporting of student 
learning outcomes, and what modifications hold promise for improvement? 


o What were the initial aims, objectives, expectations for the student learning outcomes pilot? What 
critiques have arisen? What effect has the display of learning outcomes results had on demonstrating 
accountability? 


o How has the literature, discourse, and policy environment of accountability, improvement, and 
measurement changed since the start of the pilot? How useful are the initial principles or elements 
uiding the student learning outcomes pilot? What modifications would increase its utility? 
8g g & P 


o How many institutions are involved in the student learning outcomes pilot and what is their level of 
participation (e.g., have they administered one of the tests, posted results, provided links to additional 
information)? To what extent are currently participating institutions meeting the participation agree- 
ment? 


o What do we know about other institutional quality assurance data, instruments, or material utilized by 
participating institutions to measure student learning? How would the student learning outcomes pilot 
look if it were aligned with what institutions are already doing to gather evidence of student learning 
and with what ae makers expect to address accountability concerns? 


To examine effectiveness of the student learning outcomes pilot project, the NILOA evaluation focused on the College 
Portrait as it relates to a) promoting institutional improvement of student learning and use of evidence of student 
learning; b) providing information to consumers in an understandable, meaningful, and usable format; and c) meeting 
demands of accountability. To examine the relevance of the student learning outcomes portion of the College Portrait, 
the evaluation examined how student learning outcomes information is used and understood by institutions and external 
constituents, focusing on three essential constituent groups: 


1. Jnstitutions—a) those participating in the VSA; b) those eligible but not participating; and c) those that have 
participated but no longer do so 


2. External “consumer’—guidance counselors and admissions personnel member organizations 


3. External “policy” —regional accrediting agencies, SHEEOs, NCSL, ECS, Congressional staff, higher education 
federal relations staff, and other relevant staff 


Prospective students and their families are constituent groups that were not included due to the difficulty in attaining 
access and determining a meaningful sample. Google Analytics provides some insight into the pages viewed by different 
audiences, which may well include prospective students and their families, thus, potentially serving as a proxy for infor- 
mation interest. 
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Data Collection and Analysis 


Where possible, data were analyzed for themes across multiple data sources to present an evidence-based story or repre- 
sentation of responses from surveys, focus groups, and interviews. 


Focus Groups 

A series of focus groups took place at national meetings with over 150 participants. Focus groups included representa- 
tives from VSA-participating institutions, nonparticipating institutions, and institutions that withdrew participation. 
Focus groups included representatives from the AASCU Committee on Undergraduate Education, the APLU Council 
of Presidents, the APLU Executive Committee of the Council on Academic Affairs, and government relations officers. 
Occasions for conducting focus groups included national conferences of AASCU, APLU, and the AAC&U. Focus 
groups met for roughly an hour to an hour and a half to discuss the VSA, the College Portrait, and the student learning 
outcomes pilot project. These focus groups were semi-structured in that the NILOA staff leading them posed several 
questions to prompt discussion but allowed the focus groups to explore issues that emerged. Questions addressed within 
the focus groups were drawn from the following list: 


1. What prompted your institution to participate, to decline to participate, or to withdraw from the Voluntary 
System of Accountability (VSA)? 


2. What has been your experience with the VSA? 
a. Has it met your expectations? 
b. Is the exercise practically relevant and useful? 
c. What have you learned from participating? 
d. What do you consider to be the role or purpose of the VSA? 


3. What do you see as the value of participating in the VSA when you joined? Do you think it has “taken the heat 
off” institutions? What do you consider to be the value of continuing to participate in the VSA? 


4, What changes in the student learning outcomes requirements would be instrumental to your institution 
continuing or rejoining the VSA? 

5. How have you used the College Portrait? How have you used the results of the student learning outcomes pilot 
(i.e., one of the three selected tests)? How useful are the results? 


a. Are results communicated on campus, beyond the campus, and to which groups? What was the feed- 
back or consequence? 


b. How were value-added calculations used and to what extent were these found to be useful on and off 
campus, especially with legislative, coordinating, planning boards, or the general public? 

6. Are there barriers in the student learning outcomes pilot process for institutions (e.g., putting information on 
the College Portrait, administration of the test, acquiring a representative sample of students)? 

7. What other assessment activities are you doing? Do they align with the student learning outcomes pilot require- 
ments? 

8. Do you think student learning outcomes information should be comparable across institutions? 
Should results of student learning be presented differently on the College Portrait? 

10. In what areas does the College Portrait excel regarding student learning outcomes? In what areas could it 
improve? 


11. In your opinion, what would a second generation of the student learning outcomes portion of the College 
Portrait cover, and what would it look like? 


NILOA focus group leaders took notes during and wrote summaries following the focus groups. Focus group responses 
were summarized, coded, and analyzed across groups as well as within groups to determine any emerging themes or 
trends. In the report, focus group participant responses are presented in the aggregate, as there was much consensus 
among the focus groups and no ta up trends se 3 as differing from those found across the groups. 


Interviews 

Interviews with 10 policy organization staff members and personnel from regional accreditors were conducted in person 
or via phone. Interviews lasted from 30 to 40 minutes, were semistructured, and included one or more of the following 
questions: 
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1. Are you familiar with the Voluntary System of Accountability (VSA)? Have you looked at the College Portrait, 
specifically the section on student learning outcomes? 


2. Has the VSA met your expectations for accountability and transparency regarding student learning? 

3. How important is it to accountability and transparency that evidence of student learning be comparable? In 
your opinion, is the College Portrait information comparable? 

4, In what areas does the College Portrait excel regarding student learning outcomes? In what areas could it 
improve? 


5. What more would you like to see the College Portrait be able to do? 


Interview data were summarized, coded, and analyzed to determine themes or trends. NILOA interviewers wrote 
summaries of the interviews. Themes and trends were then compared with the focus group findings for purposes of 
triangulation. 


Surveys 

Snap/flash surveys were utilized to illicit information from groups such as guidance counselors and admissions personnel, 
from institutions that withdrew from participating in the College Portrait, and from institutions that are eligible to 
participate but do not do so. Snap/flash surveys are short, 3- to 5-question surveys, generally sent via membership list- 
servs. 


Guidance Counselors 


Invitations to participate were sent to guidance counselors and admissions personnel via two national member listservs 
— one for guidance counselors and one for admissions personnel. The survey was open for three weeks, and reminder 
emails were sent at regular intervals. The survey asked the following questions: 


1. Are you familiar with the Voluntary System of Accountability (VSA), an initiative by public four-year univer- 
sities to supply relevant information to prospective students and other stakeholders through a common web- 
reporting template—the College Portrait? 


Have you looked at the College Portrait website? 
When viewing a university's College Portrait, did you look at the section on student learning outcomes? 


Was the information presented on the College Portrait clear, understandable, and usable for your purposes? 


el eS Ge TS 


Did the information on the College Portrait influence your perception of participating colleges or universities? 
If “yes,” in what ways? 


A total of two people responded to the guidance counselor and admissions personnel survey. Both respondents were 
familiar with the VSA, but neither had looked at the student learning outcomes information presented on the College 
Portrait. Outside of these two respondents, there was general confusion among guidance counselors around the VSA. 
NILOA received several phone calls and emails (from a total of 20 different guidance counselors) asking how the VSA 
differed from the Virginia Counselors Association (VCA) or if the organization we had meant to identify was the VCA 
or the VSCA (another Virginia-based school counselor association). Furthermore, there was some confusion among 
guidance counselors contacting NILOA as to if we have meant a previous organization for vocational guidance coun- 
selors that used to have a website displaying college information. Based on the general lack of response to the survey 
and the evident confusion among potential respondents, it is reasonable to suggest that the guidance counselor group is 
not familiar with or aware of the VSA and the opportunities it may provide for students seeking additional information 
about college options. 


Withdrawal Institutions 


Survey invitations were sent to the provosts of all institutions that had withdrawn from participation in the College 
Portrait. The survey was open for three weeks and one reminder e-mail was sent. The survey had a response rate of 53%, 
with 14 of the 25 institutions responding to the following questions: 


1. Did the student learning outcomes requirements of VSA influence your decision to withdraw participation 
from the College Portrait? If “yes,” what aspects of the student learning outcomes requirements were problem- 
atic? 
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2. What changes in the student learning outcomes requirements would be instrumental to your institution 
rejoining the VSA? 


3. Does your institution report results of student learning outcomes assessment online to external audiences? 


Two institutions that did not respond to all of the questions were removed from the analysis. Of the three questions’ 
possible responses, two (items 1 and 2) were open-ended text response and two were “yes/no” (first part of item 1 and 
item 3) response. The frequency of “yes/no” responses to the first part of item 1 and all of item 3 were examined using 
Microsoft Excel. All text responses were coded and examined for themes as well as their relationship to the focus group, 
interview responses, and other surveys. Due to the small number of institutions surveyed, no further analysis was 
conducted beyond frequencies. Findings from this survey are presented as representing responses from institutions that 
withdrew participation. 


Nonparticipating Institutions 


Survey invitations were sent to provosts of all institutions that were eligible to participate in the VSA College Portrait but 
that had not chosen to do so. The survey was open for three weeks, and one reminder email was sent. The survey had a 
response rate of 42%, with 72 out of 171 institutions responding to the following questions: 


1. Your institution is eligible to participate in the Voluntary System of Accountability (VSA), but does not. Please 
briefly tell us why you have chosen not to participate. 


2. If your decision not to participate in the VSA was due in part to the measurement and reporting of student 
learning outcomes requirements, what changes in this aspect of the VSA would influence your institution to 
participate? 


3. Does your institution report results of student learning outcomes assessment online to external audiences? 


Three institutions that did not respond to all of the questions or did not provide coherent, typed responses were removed 
from the data set before the data was analyzed. Of the three questions, two (items 1 and 2) were open-ended text 
response items and the third was a “yes/no” response item. All text responses were coded and examined for themes as 
well as for their relation to the focus group comments, interview responses, and data from other surveys. The frequency 
of “yes/no” responses to the third item was examined utilizing Microsoft Excel. Due to the small number of institutions 
surveyed, no further analysis was conducted beyond frequencies. In the report, findings from this survey are presented 
as representing responses from eligible nonparticipating institutions. 


Analysis of VSA Participating Institution Survey Data 

NILOA was given a copy of the responses to the VSA-administered survey of VSA-participating institutions concludin 
in August 2011. The survey was administered in such a way that multiple people within an institution could respond. 
Because several people responded to the survey at some institutions while only one person responded at others, sorting 
the data by institutional response was not possible. Nor was it possible to collapse survey responses into one institutional 
response, because responses were not always consistent between different institutional respondents. Further, because not 
all survey respondents stated their position within the institution, analyzing data based on the responding person’s posi- 
tion—such as faculty, staff, provost, or institutional researcher—was not possible. 


In total there were 248 responses to the survey, of which 39 were missing significant amounts of data, leaving 209 
responses to the survey that were utilized for analysis. Of these 209 responses, 87 were multiples, meaning that more 
than one person at an institution submitted a response. Therefore, it is important to note, the survey data discussed 
throughout this report reflects the individual responses of those who responded to the survey—not VSA-participating 
institutions as a whole. Survey responses were analyzed in SPSS only for frequency. Further statistical analyses were not 
conducted due to the questionable nature of some of the responses. In some instances, respondents did not complete 
two-part questions, for example, 1) if an assessment activity was utilized at an institution, 2) was the activity found to 
be useful to the institution. On questions where respondents were able to rate the usefulness of additional assessment 
options, data were analyzed based on affirmative responses to the first part of the question—as it is unlikely that a 
respondent who had not used an assessment activity could accurately respond to a question about the activity’s overall 
usefulness for the institution. The analyses and findings from this survey are identified in this report as those from the 
VSA-participating institutions that responded to the VSA-administered survey. These findings were compared with the 
NILOA-administered surveys, focus group responses, interview responses, and what NILOA researchers found on insti- 
tutional College Portraits. 
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i a Analyt 


Google permny sesoese information about website traffic in terms of the number of visitors to a specific page as well 
as the average time users spend on a page. Using time spent on a page as a proxy for interest, NILOA examined Google 
Analytics from two time periods: the first from October 2009 through August 2010 and the second spanning February 
2011 to December 2011. The time periods were divided based on all available analytics up to December 5, 2011 when 
the College Portrait website changed to the new format. The break in Google Analytics from August 2010 to February 
2011 was due to the analytics data not being available from the College Portrait site. The analytics were examined 
specifically for user activity on the student learning outcomes page of the College Portrait as well as generally across the 
College Portrait site as a whole. Individual College Portraits in which any user viewed the student learning outcomes 
information were included for analysis in both time periods. From each of the selected time periods, we exported all the 
pages viewed from Google Analytics into Excel spreadsheets for further analysis. Each individual institution’s College 
Portrait main page was matched with their student learning outcomes page on unique pages views, bounces, and time 
spent on site. From this match, we were able to compute for each individual College Portrait as well as across College 
Portraits the percentage of viewers who went to an institutional College Portrait main page and then clicked on the link 
to the institution’s student learning outcomes page. Time spent on the student learning outcomes page was figured as an 
average across all student learning outcomes pages viewed in each time period. 


In addition to the individual College Portrait pages and views, we analyzed the top 500 most-viewed pages on the entire 
College Portrait website. For the 2009-2010 period, two of these were the student learning outcomes pages of individual 
institutions.’ In 2011, no student learning outcomes page made the top 500, but 90% of the top 500 pages viewed 
were cost-estimator pages. Also, using Google Analytics, we examined traffic to the entire College Portrait. Through the 
existing analytic structure we looked at the unique page views of the entire College Portrait website in 2011 to determine 
which pages users were visiting. Data across the entire College Portrait website for 2011 is included in this report. 


Several focus group respondents mentioned the additional California Public Good page in the College Portrait as an 
additional means to provide information to consumers. We pulled the Google Analytics for activity on this page into 
a Microsfot Excel file from two time periods: the first from February 2011 through December 2011 and the second 
covering a one-month period between January and February 2012. This was done to compare numbers from each time 
period to ensure we had an accurate picture of traffic. Since the California Public Good page became available, 9% of 
users who visited a participating California institution’s College Portrait main page also went to the institution’s Public 
Good page and spent an average of one minute there. While a low frequency overall, this was higher, by about 2%, than 
the frequency of user visits to student learning outcomes pages. 


College Portrait Statistics 

College Portrait statistics were analyzed using data in two formats: 1) a copy of a snapshot of the College Portrait data- 
base; and 2) a web scan of College Portrait sites in early February 2012. The VSA provided NILOA with a copy of its 
College Portrait database as it related to the student learning outcomes information up to November 2011. This infor- 
mation included the year an institution joined and whether results of a standardized test, along with the name of the 
test, were posted on the institution’s College Portrait. Database information was analyzed for frequency of institutional 
compliance with the participation agreement. 


In addition to the database information, the institutional College Portrait of every individual participating institution 
was examined, at www.collegeportrait.com, using the map feature to determine state by state which institutions had 
active College Portraits and which did not. Individual College Portrait student learning outcomes pages were exam- 
ined to see if institutions were meeting the VSA Participation Agreement—which states that institutions should post 
results of one of the three standardized tests, should include an institutional link to additional institutional information 
on assessment of student learning, and may provide a narrative on institutional assessment activities. Compliance or 
noncompliance with each of these three parts of the VSA Participation Agreement was captured in a Microsoft Excel 
spreadsheet, and frequencies as well as content were analyzed. 


1 The two student learning outcomes pages were for Clemson University (ranked 355) and the University of Maryland at College Park (ranked 
386). 
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